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We evaluate the extent to which ePortfolios can be used to assess applied and collaborative learning 
and academic identity among community college students from underrepresented minority groups 
who participated in a summer research program. Thirty-eight students were evaluated by their 
research sponsor and two or three naive faculty evaluators. Faculty sponsors evaluated students 
based on personal interactions and the students’ ePortfolios. Naive faculty evaluated students using 
only the ePortfolios. We found: (1) The rubrics designed to assess applied and collaborative learning 
and academic identity had good internal consistency, (2) naive evaluators found some evidence of all 
learning outcomes, (3) faculty sponsors found evidence of more learning outcomes than naive 
evaluators, and (4) individual ePortfolios varied in the extent to which they documented learning 
outcomes. We conclude that ePortfolios can be used as a reliable means of documenting applied and 
collaborative learning and academic identity. 


Research experiences help students develop an 
academic identity that increases underrepresented 
minority (URM) student persistence in STEM 
disciplines (Jackson, Starobin, & Laanan, 2013). 
Participation in research has also been found to improve 
the persistence of URM women who begin their studies 
at community colleges (Jackson et al., 2013) and is 
associated with higher levels of perceived support and 
academic persistence among all students from 
underrepresented groups (Barlow & Villarejo, 2004; 
Maton & Hrabowski, 2004; Yelamarthi & Mawasha, 
2008). The ability to document learning that occurs in 
the context of research experiences would be useful for 
both research purposes and efforts to credit prior 
learning. In the current paper we describe the 
evaluation of our efforts to use ePortfolios created 
during a research program to assess applied learning 
and academic identity in URM community college 
students. By comparing assessments made by faculty 
who worked directly with students to those made by 
faculty who were unfamiliar with the students, we are 
able to evaluate the extent to which ePortfolios 
document actual learning. 

As more learning is taking place outside traditional 
higher education settings, learners need means of 
documenting their knowledge and skills (Travers, 2012). 
Portfolio review (both electronic and traditional) is 
commonly used by academic institutions to evaluate prior 
learning (Conrad & McGreal, 2012). Prior learning 
assessments are relatively high stakes and should afford 
students the opportunity to present their learning fully 
(Stenlund, 2013). ePortfolios allow students to include a 
variety of documents along with reflection, providing a 
rich account of the learning that has taken place (Travers, 
2012). The opportunity for students to provide context for 
their work is especially important because many 
institutions review only the work products, without having 
any contact with the student. 


We focus on assessment because we believe that 
this constitutes a unique contribution to the existing 
literature on ePortfolios. Despite reasonable theoretical 
justification for using ePortfolios, there is little 
empirical evidence for their effectiveness. In their 
review of the empirical literature, Bryant and Chittum 
(2013) found that only 15% of peer-reviewed articles 
addressed student outcomes, and only half of those 
articles assessed academic learning outcomes 
specifically. Bryant and Chittum (2013) argued that 
researchers should assess the extent to which 
ePortfolios are linked to student learning outcomes, 
especially in STEM disciplines. Rhodes, Chen, Watson, 
and Garrison (2014) also pointed to the need for more 
rigorous research on the benefits of ePortfolios. 

Our examination of ePortfolio use was conducted 
in the context of a residential summer research 
program. Since 2000, Purchase College of the State 
University of New York has offered the Baccalaureate 
and Beyond program to support URM students as they 
transition from community colleges to four-year 
institutions. Each year, the program serves 
approximately 25 students from six community 
colleges. Students work full-time in small groups 
conducting original research in biology, chemistry, 
computer science, environmental science, neuroscience, 
or psychology. 

In the current work, we elected to use the applied 
and collaborative learning proficiency from the Degree 
Qualifications Profile (DQP) as the basis for our 
assessment. The DQP was developed to provide a 
complete description of the proficiencies that students 
should obtain in higher education. The DQP purpose 
has been described as “what students should know and 
be able to do once they earn their degrees—at any level, 
in any field of study” (Adelman, Ewell, Gaston, & 
Geary Schneider, 2014, p. 7). As such, the DQP is well 
suited to assessing students who are transitioning 
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between community college and a four-year school. 
Applied and collaborative learning, one of the five 
areas of proficiency outlined in the DQP, includes 
research and creative activities that involve “innovation 
and fluency in addressing unscripted problems in 
scholarly inquiry, at work and in other settings outside 
the classroom” (Adelman et ah, 2014, p. 6). At the 
bachelor’s level, applied and collaborative learning 
includes four areas that are present in summer research 
programs: (1) presents a project, (2) negotiates a 
strategy for group research, (3) writes a design, and (4) 
completes a substantial project (Adelman et ah, 2014). 
Because applied and collaborative learning is the 
proficiency that is most likely to occur in non- 
traditional settings (Adelman et ah, 2014), its 
assessment can present a challenge. ePortfolios have the 
potential to serve as a useful tool in the reliable 
assessment of applied and collaborative learning. 

ePortfolios are useful as a means of documenting 
learning from non-traditional activities such as research 
experiences (Wang, 2009) and are hypothesized to 
support reflection, engagement, and active learning 
(Yancey, 2009). Accordingly, ePortfolio use in higher 
education has become increasingly prevalent (Rhodes, 
et ah, 2014). There is evidence that ePortfolios help 
students and faculty evaluate growth and reflect on 
students’ academic achievements (Buzzetto-More, 
2010). Eynon, Gambino, and Torok (2014) found 
evidence that ePortfolio use correlates positively with 
student success indicators (course pass rates, GPA, and 
retention rates) and can help advance and support deep 
thinking, integration, and personal growth. The creation 
of ePortfolios has also been found to help students 
develop a sense of academic identity, future orientation, 
and belonging to a community of scholars (Nguyen, 
2013; Singer-Freeman, Bastone, & Skrivanek, 2014). 

Unfortunately, to date, much of the research examining 
ePortfolio use as an assessment tool has focused on student 
and faculty attitudes about ePortfolios rather than on the 
usefulness of ePortfolios as a means of reliable assessment 
(Rhodes et ah, 2014). For example, Curtis and Wu (2012) 
reported that healthcare educators have become more 
accepting of ePortfolio use as an assessment tool. Similarly, 
Garrett, MacPhee, and Jackson (2013) found that nursing 
faculty considered ePortfolio evaluation an effective means 
of assessment. When ePortfolios were used as the primary 
means of assessment in a Child Development class, 84% of 
students felt they encouraged reflection, 77% felt they 
provided a permanent record of learning, and 87% felt their 
use in the class should be continued (Singer-Freeman & 
Bastone, 2015). Bryant and Chittum (2013) cautioned that 
attitudes are not necessarily a good indicator of usefulness 
as a learning tool. 

Additional evidence for the usefulness of 
ePortfolios is provided by Buyarski and Landis (2014), 
who examined the efficacy of using ePortfolios to 


assess learning in a first-year seminar. Using rubrics, 
along with examination of students’ narratives, trained 
faculty unfamiliar with the students whose work they 
assessed were able to provide reliable evidence of 
learning. We adapt Buyarski and Landis’s (2014) 
methodology to examine the efficacy of rubrics as a 
means for assessing applied and collaborative learning 
and academic identity in ePortfolios. 

In a review of research examining the value and 
educational consequences of rubric use, Jonsson and 
Svingby (2007) found that analytic, topic-specific 
rubrics were most useful in assessment of student 
performance. In general, rubric use makes assessment 
by multiple evaluators more reliable. Jonsson and 
Svingby (2007) reported that inter-rater reliability rates 
ranged from 4-100%, with the majority falling in the 
range of 55-75%. In general, reliability rates were lower 
in instances in which tasks or products were not 
uniform. More extensive rater training was generally 
associated with higher levels of reliability. However, 
high reliability was not necessarily associated with high 
validity. Validity of rubrics varies widely and depends 
on the care with which rubrics are developed to align 
with a construct of interest. Rubrics also enhance 
learning and instruction by making expectations clear to 
both students and faculty. Accordingly, a good rubric 
that is in alignment with a construct might guide 
instruction to align more fully with the construct. 

In the current work, we assessed whether evaluators 
could reliably use an ePortfolio to assess applied and 
collaborative learning and academic identity. We utilized 
two sets of evaluators, with different levels of direct 
knowledge of the students, in order to gain insight into 
the extent to which ePortfolios capture authentic 
learning. Faculty who directly supervised students in the 
research lab (faculty sponsors) evaluated students using 
their complete knowledge of the student from personal 
interactions as well as from the student’s ePortfolio. 
Faculty unfamiliar with the student are referred to as 
naive evaluators because they assessed learning using 
only the ePortfolio and were naive with regard to 
performance in the research lab. We hypothesized that 
naive evaluators would see less evidence of learning than 
faculty sponsors because they had less information. 
However, should naive evaluators report stronger 
evidence of learning than faculty sponsors, this would 
raise the possibility that students might be able to inflate 
their proficiency artificially in an ePortfolio. We 
hypothesized that ePortfolios would document applied 
and collaborative learning and academic identity, but 
that direct faculty knowledge of students would 
provide more robust evidence than ePortfolios alone. 
We were also interested in assessing the extent to 
which the created rubrics were valid and reliable 
measures of applied and collaborative learning and 
academic identity. 
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Method 

Participants 

Students. The summer research program included 
22 students in 2013 and 22 students in 2014. Four 
students who participated in the research program in 
2013 were excluded due to the death of their research 
sponsor, and two students were excluded due to failure 
to share their ePortfolios. This resulted in a sample of 
38 students (29 females and 9 males). Our sample 
included 24 students who identified as African 
American, 10 who identified as Latino, two individuals 
who reported mixed African American and Latino 
ancestry, one who identified as Asian, and one who 
identified as Native American. Fifteen students 
completed research in psychology or neuroscience, 13 
in biology, seven in biochemistry, and three in 
environmental science. 

Faculty sponsors. Eight faculty served as research 
sponsors: Four of these were sponsors in both 2013 and 
2014, two sponsored students only in 2013, and two 
sponsored students only in 2014. All sponsors had 
PhDs in a STEM discipline and were full-time faculty 
members at the college. There were five assistant 
professors, two associate professors, and one full 
professor. There were five males and three females. All 
faculty identified as White. 

Naive evaluators. We decided to have faculty 
sponsors evaluate both the students they sponsored and 
students with whom they were unfamiliar to ensure that 
similar standards would be used for both sets of 
evaluations. When rating unfamiliar students, faculty 
are referred to as naive evaluators. Three additional 
STEM faculty who did not sponsor students served as 
naive evaluators. They all had PhDs and were full-time 
faculty members (one lecturer, one associate professor, 
and one full professor). There were two females and 
one male, and all identified as White. 

Materials and Procedure 

Learning outcome and rubric development. To 

identify expected proficiencies, six faculty research 
sponsors individually created lists of learning outcomes 
that were then discussed in a focus group with the 
authors and two outside experts in rubric construction. 
The group reached consensus on the desired learning 
outcomes and the products that could be used to 
evaluate mastery of these outcomes. The outcomes 
included items associated with both applied and 
collaborative learning and academic identity (see Table 
1). The applied and collaborative learning outcomes 
were brought into alignment with the applied and 
collaborative learning proficiency from the DQP 
(Adelman et al., 2014). Rubrics were developed to 


clarify expectations for each of the learning outcomes. 
The six faculty sponsors piloted the rubrics, using them 
to assess 20 ePortfolios. Results of this pilot work and a 
focus group with the participating faculty revealed that 
the ePortfolios did not include sufficient information 
for assessment of the quality of revisions or the 
development of collaborative strategies. Accordingly, 
these learning outcomes were removed (see Figures 1 
and 2). 

ePortfolio creation. We instructed students to 
create ePortfolios that would document their summer 
experiences. Students were provided with the expected 
learning outcomes and products and shown how to 
create and share pages. Work on the ePortfolios 
occurred after the students’ work day was complete. 
Students created their ePortfolios independently, 
without direct supervision from the research sponsor. 
Program staff held weekly workshops in which the 
students were required to contribute a minimum of one 
journal entry, one image that documented learning in 
some area, and one piece of writing that documented 
learning. Students were required to provide a caption 
for each image that explained how it documented 
learning. Additionally, students were required to 
respond each week to a specific written prompt (see 
Appendix). Students were not provided with a template. 
Instead, staff worked individually with students to 
develop content and design the ePortfolio. The 
ePortfolios were not graded. However, program faculty, 
staff, and students provided comments on the 
ePortfolios that individual students shared with the 
group. 

Evaluator training. Faculty sponsors were 
instructed on the use of the rubrics during a faculty 
meeting that took place one week before the faculty 
were to begin evaluating. Two naive evaluators who 
were not faculty sponsors attended the group instruction 
session. The remaining naive evaluator who was not a 
faculty sponsor received individual instruction. During 
the training, evaluators reviewed the rubrics and 
discussed possible products that could be used to 
document proficiency. Evaluators were instructed to 
read reflective writing carefully as a source of 
information about academic identity proficiencies. The 
evaluators completed the rubrics by selecting from five 
possible ratings: exceeds expectations, meets 
expectations, approaches expectations, does not meet 
expectations, and cannot evaluate. All 38 students were 
assessed by their respective research sponsors. Twenty- 
one students were assessed by three naive evaluators 
and 17 students were assessed by two naive evaluators. 
We elected not to have faculty reach consistency on 
sample ePortfolios because we were interested in the 
rubrics’ utility in a minimal training environment. 
Although research has established higher inter-rater 
reliability with practice sessions, this sort of training is 
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Table 1 


Initial Learning Outcomes Generated by Faculty Research Sponsors 


Construct 

Learning outcome 

Evidence 

Academic identity 

Identifies hopes and goals for experience 
Demonstrates confidence sharing ideas 

Engages with the research 

Identifies learning from experience 

Constructs plans for academic future 

Refines ideas about possible careers 

Journal entries 

Interim and final report 

Applied and 
collaborative learning 

Literature 

Summarizes research literature 

Articulates contribution to existing knowledge 

Annotations 

Interim and final report 


Research Design 

States project goals 

Articulates research hypothesis 

Describes research design 

Interim and final report 

Journal entries 


Data Collection 

Successfully implements methodology 

Documents data collection 

Experimental results 

Interim and final report 

Journal entries 


Data Analysis 

Organizes data 

Performs calculations correctly 

Draws appropriate conclusions 

Communicates results 

Experimental results 

Interim and final report 


Collaboration 

Demonstrates collaboration skills 

Works with team to draft research abstract 

Works with team to draft final presentation 

Interim and final report 

Journal entries 


Revisions 

Uses faculty feedback to revise work 

Abstract 

Interim and final report 

Conference submissions 


Oral Presentation 

Presents work orally with confidence and clarity 

Final report 


not associated with improved validity (Jonsson & 
Svingby, 2007). Additionally, in real world 
applications, rubrics are frequently used by individuals 
who have not received reliability training. 

Results 

Coding 

Because our primary goal was to determine 
whether ePortfolios could provide reliable information 
that would enable ePortfolio-based assessment of prior 
learning and academic identity, we focused on whether 


evaluators reported that a learning outcome had been 
met (grouping meets expectations and exceeds 
expectations ) or not met (grouping approaches 
expectations, does not meet expectations , and cannot 
evaluate). We treated cannot evaluate as an indication 
that a learning outcome had not been met because this 
response was given in instances in which material 
related to an outcome was not present in the ePortfolio. 

Inter-Item Reliability 

To determine inter-item reliability we calculated 
Cronbach’s alphas for ePortfolio-based assessment of 
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Figure 1 

Academic Identity Rubrics 



Exceeds expectations 

Meets expectations 

Approaches 

expectations 

Does not meet 
expectations 

Hopes and 
goals 

Provides a fully 
developed discussion 
of goals for experience. 

Provides a good 
discussion of goals 
for experience. 

Identifies some goals 
for experience. 

Does not identify 
goals for experience. 

Confidence 
sharing ideas 

Demonstrates 
confidence in sharing 
intellectual ideas. 

Demonstrates some 
confidence in sharing 
intellectual ideas. 

Demonstrates limited 
confidence in sharing 
intellectual ideas. 

Does not demonstrate 
confidence in sharing 
intellectual ideas. 

Engagement 
with research 

Notes indicate full 
engagement with the 
research process. 

Notes indicate good 
engagement with the 
research process. 

Notes indicate some 
engagement with the 
research process. 

Notes indicate limited 
or no engagement 
with the research 
process. 

Learning from 
experience 

Offers fully developed 
insights into learning 
gained from 
experience. 

Offers good insights 
into learning gained 
from experience. 

Offers some insights 
into learning gained 
from experience. 

Does not identify 
learning gained from 
experience. 

Careers 

Shares fully developed 
ideas about possible 
careers. 

Shares somewhat 
developed ideas 
about possible 
careers. 

Shares poorly 
developed ideas about 
possible careers. 

Does not share ideas 
about possible 
careers. 

Academic 

liiture 

Shares fully developed 
plans for academic 
liiture. 

Shares somewhat 
developed plans for 
academic future. 

Shares poorly 
developed plans for 
academic future. 

Does not share plans 
for academic future. 


each construct. We only included ePortfolio-based 
ratings in these calculations because we were interested 
in the use of the rubrics as measures of proficiencies 
demonstrated by the ePortfolio and not as measures of 
proficiencies demonstrated by direct knowledge of the 
student. We observed alphas of .78 for applied and 
collaborative learning and .69 for academic identity. 

Inter-Rater Reliability 

To determine the level of agreement between 
faculty sponsors and naive evaluators, we calculated the 
reliability between faculty sponsors and naive 
evaluators responding to the same student. We found 
that the average sponsor-evaluator reliability was 0.45 
(SD = 0.19). This low reliability score is consistent with 
our presupposition that sponsors would have available 
to them substantially more information than naive 
evaluators when evaluating students. 

To determine whether naive evaluators were 
assessing the ePortfolios similarly, we calculated the 
reliability between naive evaluators responding to the 
same ePortfolio. We excluded the scores provided by 
the single naive evaluator with a reliability score of 
0.42 because this is outside the typical reliability range 


reported by Jonsson and Svingby (2007). With this 
naive evaluator excluded, the remaining naive 
evaluators had reliability scores of between 0.62 and 
.72. In instances in which there were only two naive 
evaluators, reliability was 0.71 (SD = 0.14), with 
reliability scores ranging from 0.50 to 0.93. In instances 
in which three evaluators assessed a single ePortfolio 
we calculated the reliability between the two evaluators 
who had the highest level of agreement. We found that 
the average reliability was 0.86 (SD = 0.11), with 
scores ranging from 0.64 to 1.00. 

Individual Learning Outcomes 

Tables 2 and 3 report the number and percentage of 
instances in which sponsors and naive evaluators 
credited each learning outcome. Because two or three 
naive evaluators assessed each ePortfolio, there were a 
total of 90 naive evaluator assessments. As can be seen 
in Tables 2 and 3, faculty sponsors were aware of 
learning that was not evident to naive evaluators, 
explaining the low levels of sponsor-evaluator 
reliability reported above. Faculty sponsors credited 
between 47% and 87%, and naive evaluators credited 
between 7% and 66%, of individual outcomes. In fact, 




Singer-Freeman, Bastone, & Skrivanek 


Applied and Collaborative Learning 50 


Figure 2 

Applied and Collaborative Learning Rubrics 



Exceeds expectations 

Meets expectations 

Approaches 

expectations 

Does not meet 
expectations 

Understands 

literature 

Demonstrates an 
excellent 

understanding of the 
literature. 

Demonstrates a good 
understanding of the 
literature. 

Demonstrates some 
understanding of the 
literature. 

Does not demonstrate 
understanding of the 
literature. 

Analyzes 

literature 

Very effectively 
analyzes literature. 

Effectively analyzes 
literature. 

Offers limited 
analysis of literature. 

Does not analyze 
literature. 

Project goals 

Demonstrates 

excellent 

understanding of the 
project significance. 

Demonstrates a good 
understanding of the 
project significance. 

Demonstrates some 
understanding of the 
project significance. 

Does not demonstrate 
understanding of the 
project significance. 

Project 

hypothesis 

Demonstrates an 
excellent 

understanding of the 
research hypothesis. 

Demonstrates a good 
understanding of the 
research hypothesis. 

Demonstrates some 
understanding of the 
research hypothesis. 

Does not demonstrate 
understanding of the 
research hypothesis. 

Research design 

Demonstrates an 
excellent 

understanding of the 
project research 
design. 

Demonstrates a good 
understanding of the 
project research 
design. 

Demonstrates some 
understanding of the 
project research 
design. 

Does not demonstrate 
understanding of the 
project research 
design. 

Data collection 

Collects and records 
data with no errors. 

Collects and records 
data with very few 
errors. 

Collects and records 
data with some 

errors. 

Does not collect and 
record data. 

Data analysis 

Fully understands the 
analyses. 

Generally understands 
the analyses. 

Understands some of 
the analyses. 

Does not understand 
the analyses. 

Draws 

conclusions 

Fully understands the 
relation between 
results and hypothesis. 

Generally understands 
the relation between 
results and 
hypothesis. 

Understands some of 
the relation between 
results and 
hypothesis. 

Does not understand 
the relation between 
results and 
hypothesis. 


Table 2 


Evaluations Crediting Academic Identity Outcomes 


Learning outcome 

Faculty sponsor 

Na'ive evaluator 

n 

% 

n 

% 

Hopes and goals 

18 

47% 

27 

30% 

Confidence sharing ideas 

31 

82% 

59 

66% 

Academic future 

31 

82% 

43 

48% 

Careers 

33 

87% 

50 

56% 

Learning from experience 

27 

71% 

43 

56% 

Engagement with research 

32 

84% 

50 

56% 


Note. Faculty sponsor n = 38. Naive evaluator n = 90. 


for all outcomes assessed, faculty sponsors reported 
higher rates of acquisition than naive evaluators. 
Differences between faculty sponsor and naive 
evaluator assessments of individual learning outcomes 
ranged from 15% (learning from experience) to 69% 
(data collection). 

The naive evaluators reported that ePortfolios 
included evidence of academic identity outcomes 


between 27% and 59% of the time and included 
evidence of applied and collaborative learning 
outcomes between 7% and 41% of the time. Although 
the evidence for individual applied and collaborative 
learning outcomes was low, because there were eight 
unique outcomes, students could show evidence of 
applied and collaborative learning without having 
provided evidence of every learning outcome. Naive 
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Table 3 


Evaluations Crediting Applied and Collaborative Learning Outcomes 


Learning outcome 

Faculty sponsor 

Na'ive evaluator 

n 

% 

n 

% 

Understands literature 

25 

66% 

20 

22% 

Analyzes literature 

21 

55% 

15 

17% 

Project goals 

32 

84% 

37 

41% 

Project hypothesis 

33 

87% 

32 

36% 

Research design 

29 

76% 

29 

32% 

Data collection 

29 

76% 

6 

7% 

Data analysis 

22 

58% 

22 

24% 

Draws conclusions 

20 

53% 

20 

22% 


Note. Faculty sponsor n = 38. Naive evaluator n = 90. 


evaluators agreed that five (13%) of the 38 ePortfolios 
failed to show evidence of any applied and 
collaborative learning. However, the faculty sponsors of 
these students reported evidence of between four and 
eight of the applied and collaborative learning 
outcomes. We conclude that these five students 
demonstrated applied and collaborative learning but 
failed to document mastery in their ePortfolios. 

ePortfolio Capture Rates 

To determine the extent to which naive evaluators 
were aware of students’ mastery of learning outcomes, 
we limited our sample for each learning outcome to 
students who had been credited by their faculty 
sponsor. These numbers are reported in Tables 4 and 5. 
We then calculated the number of times at least one 
naive evaluator credited each of these students in order 
to determine the percentage of times naive evaluators 
credited learning that had been credited by sponsors. 
We believe that these percentages are the best measure 
of the extent to which naive evaluators were able to see 
evidence of actual learning. We will refer to this 
measure as ePortfolio capture. 

As can be seen in Table 4, the ePortfolio capture 
rates for academic identity outcomes were excellent, 
with rates ranging from 67%-87%. However, as can be 
seen in Table 5, ePortfolio capture rates for applied and 
collaborative learning outcomes were lower, ranging 
from 14%-67%. Capture rates were over 50% for 
project goals, project hypothesis, and research design. 
Capture rates ranging from 30%-40% were observed 
for understands literature, analyzes literature, data 
analysis, and draws conclusions. The lowest capture 
rate of 14% was observed for data collection. 

Differences in Sponsor and Naive Evaluator Ratings 

Another way to determine whether naive 
evaluators can reliably assess student learning is to 
compare the frequency with which faculty sponsors 


credited students with learning that the majority of 
naive evaluators did not to the frequency with which 
the majority of naive evaluators credited a student with 
learning that the faculty sponsors did not. Because the 
faculty sponsors had the benefit of both ePortfolio 
review and personal knowledge, we expected there 
would be more instances in which sponsors credited 
learning than the reverse. We found that there were far 
more instances in which sponsors credited students with 
learning when the naive evaluators did not (18%-58%) 
than there were instances in which naive evaluators 
credited students with learning when the sponsors did 
not (0%-11%). The most common applied and 
collaborative learning outcomes credited by naive 
evaluators but not sponsors were draws conclusions 
(11%), project goals (8%), and research design (8%). 
The most common academic identity outcomes that 
were credited by naive evaluators but not faculty 
sponsors were confidence sharing ideas (11%) and 
plans for academic future (8%). 

Differences between Students 

Although all of the students who participate in our 
program are enrolled in community colleges when they 
apply, some students enter our program ready to attend 
a 4-year school while others plan to return to 
community college. We divided our sample into 
students who would be attending a 4-year institution 
after completion of the summer program (n =16) and 
students who would be returning to their two-year 
institution (n = 22). We hypothesized that more 
advanced students might have been better able than less 
advanced students to master (or document) applied and 
collaborative learning at the bachelor’s level. Table 6 
reports the average number of learning outcomes 
associated with applied and collaborative learning and 
academic identity as a function of academic status and 
evaluator. 

To investigate the effects of academic status and 
evaluator on applied and collaborative learning we 
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Table 4 


Academic Identity Outcomes: Frequency of Naive Evaluator Credit for Outcomes Credited by Faculty Sponsor 


Learning outcome 

Number credited by sponsor 

No. and % credited by naive evaluator 

Hopes and goals 

18 

12 (67%) 

Confidence sharing ideas 

31 

27 (87%) 

Academic future 

31 

21 (68%) 

Careers 

33 

23 (70%) 

Learning from experience 

27 

20 (74%) 

Engagement with research 

32 

27 (84%) 


Table 5 

Applied and Collaborative Learning: Frequency of Naive Evaluator Credit for Outcomes Credited by Faculty Sponsor 

Learning outcome 

Number credited by sponsor 

No. and % credited by naive evaluator 

Understands literature 

25 

10 (40%) 

Analyzes literature 

21 

8 (38%) 

Project goals 

32 

20 (63%) 

Project hypothesis 

33 

22 (67%) 

Research design 

29 

15 (52%) 

Data collection 

29 

4 (14%) 

Data analysis 

22 

8 (36%) 

Draws conclusions 

20 

6 (30%) 


Table 6 

Average Number of Learning Outcomes Associated with Applied and Collaborative Learning and Academic Identity 

Reported as a Function of Academic Status and Evaluator 

Learning outcome 

Evaluator 

Community 
college students 
M(SD) 

4-year college 
students 
M(SD) 

All students 

M (SD) 

Applied and collaborative 

Faculty sponsor 

5.41 (2.40) 

5.75 (2.05) 

5.58 (0.35) 

learning (8 items) 

Naive evaluators 

1.61 (1.82) 

2.77 (2.57) 

2.19 (0.24) 

Academic identity (6 items) 

Faculty sponsor 

4.23 (0.37) 

4.94 (0.44) 

4.02 (0.27) 


Naive evaluators 

2.98 (0.23) 

3.01 (0.32) 

3.61 (0.22) 


calculated a 2 (academic status: 2-year school, 4-year 
school) x 2 (Evaluator: faculty sponsor, naive 
evaluator) ANOVA on the total number of learning 
outcomes credited. We observed a significant and 
strong effect of evaluator, F(\, 127) = 63.11,/? < .001, 
rj p = .33, such that faculty sponsors reported more 
evidence of learning (M = 5.58) than naive evaluators 
(M = 2.19). We also observed a marginally significant 
effect of academic status, F( 1, 127) = 3.11, = .08, rj p 2 

= .03, such that more advanced students were credited 
with higher levels of applied and collaborative learning 
(m = 4.26) than less advanced students (m = 3.51). We 
failed to observe an interaction. 

To investigate the effects of Academic Status and 
Evaluator on Academic Identity we calculated a 2 
(Academic Status: 2-year school, 4-year school) x 2 
(Evaluator: faculty sponsor, naive evaluator) ANOVA 
on the total number of academic identity outcomes 


credited. We observed a significant effect of Evaluator, 
F( 1, 127) = 19.74 ,p < .001,= .14, such that faculty 
sponsors reported more evidence of identity (M= 4.58) 
than naive evaluators (M = 3.04). We failed to observe 
an effect of academic status or an interaction between 
academic status and evaluator. 

Discussion 

In the current work, we describe our use of rubrics 
to assess applied and collaborative learning and 
academic identity in ePortfolios that were created 
during a summer research program for community 
college students from underserved groups. Faculty 
sponsors evaluated individual students using their 
knowledge from personal interactions as well as the 
student’s ePortfolio. Naive evaluators evaluated 
students using only the ePortfolio. As hypothesized, we 
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found evidence that ePortfolios document applied and 
collaborative learning and academic identity. We also 
found that the rubrics we designed to assess these 
constructs appear to be reliable measures of the 
constructs under investigation. 

The rubrics were developed following practices 
that support their validity. Experienced mentors who 
were familiar with the research program and its goals 
worked with assessment experts to develop and refine 
the rubrics over a year during which they used them to 
assess students. Construct reliability is supported by the 
adequate levels of inter-item similarity obtained for 
each measure. We observed variations in the quality of 
individual naive evaluators’ work. The range of quality 
was demonstrated by the improvement in inter¬ 
evaluator reliability when three evaluators assessed a 
single student and the two with the highest levels of 
agreement were included in the calculation. 
Nonetheless, the observed levels of inter-rater 
reliability are well within the range of reliability 
generally seen with rubric use (Jonsson & Svingby, 
2007). We conclude that these rubrics are valid and 
reliable tools that might be used in the assessment of 
student ePortfolios. 

As expected, faculty sponsors credited students 
with more learning outcomes than naive evaluators. We 
believe that this level of disagreement should be 
interpreted as evidence of the rubrics’ sensitivity. An 
effective rubric should differentiate between the 
learning evidenced in an ePortfolio alone and the 
learning evidenced in many hours of shared work in 
addition to an ePortfolio. This interpretation is 
supported by the finding that, for every measured 
learning outcome, faculty sponsors gave credit more 
frequently than naive evaluators. Similarly, faculty 
sponsors gave credit when naive evaluators did not 
more frequently than naive evaluators gave credit when 
faculty sponsors did not. 

The few instances in which naive evaluators gave 
credit when faculty sponsors did not may provide 
insight into the standards employed by the two types of 
evaluators. It is possible that faculty sponsors have 
higher standards of proof than naive evaluators. We 
believe this is likely to have affected the evaluation of 
the applied and collaborative learning outcomes. The 
applied and collaborative learning outcomes that were 
most often credited by naive evaluators but not faculty 
sponsors were draws conclusions, project goals, and 
research design. For each of these learning outcomes, 
two or three students received credit from naive 
evaluators but not faculty sponsors. These three 
outcomes are the most conceptually difficult applied 
and collaborative learning outcomes and the most 
specific to the individual research group. Accordingly, 
faculty sponsors may have been somewhat less lenient 
than naive evaluators in crediting these outcomes. 


We were primarily interested in determining the 
extent to which actual learning could be assessed via 
ePortfolios. Naive evaluators credited individual 
students with 2.19 outcomes on average (out of eight 
possible). On the basis of this finding we conclude that, 
in general, naive evaluators require a high standard of 
proof before credit is awarded. Additionally, when we 
limited our sample for each learning outcome to 
students who had been credited with the outcome by 
their faculty sponsor, we observed relatively high 
ePortfolio capture rates (30%-67%) for all individual 
learning outcomes except data collection (14%). We 
believe that high ePortfolio capture rates indicate that 
naive evaluators were able to credit actual learning. 

Individual ePortfolios varied in their quality and 
completeness. We found that there was a trend in which 
students who were ready to transfer to four-year schools 
demonstrated evidence of more applied and 
collaborative learning outcomes than students who were 
planning on returning to community colleges. Given 
that we assessed applied and collaborative learning at 
the bachelor’s level, it is not surprising that students 
who were prepared to transfer demonstrated more 
proficiency than those who were not yet ready to 
transfer. This finding can be interpreted in several 
ways. It is possible that more advanced students learned 
more than less advanced students. However, it is also 
possible that students who were about to transfer to a 4- 
year school were more able or more motivated to 
document their learning than students returning to 
community college. 

We found that, regardless of academic level, 
students were similarly able to document academic 
identity in their ePortfolios. We believe that this is an 
important finding. In previous work we have argued 
that the development of academic identity in summer 
research programs may be a central element that leads 
to increased academic persistence (Singer-Freeman et 
ah, 2014). Students with an enhanced sense of 
academic identity who return to community college 
may be more likely to transfer to a four-year school in 
the future and those who go on to four-year schools 
may be more likely to complete their bachelor’s 
degrees. 

Limitations and Implications 

This research occurred in an applied setting. 
Consequently, it was subject to lower levels of control 
than a more structured experiment would be. We 
elected to provide relatively little training to our 
evaluators. By examining the use of the rubrics without 
extensive training we were able to determine the extent 
to which the rubrics themselves enabled reliable 
evaluations. We found that levels of inter-rater 
reliability were within ranges reported elsewhere. 
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Nonetheless, if these rubrics were to be used with the 
purpose of crediting prior learning, training on sample 
ePortfolios would be advisable. If such training is not 
conducted, using three naive evaluators would be 
preferable to using two. 

Because this research was conducted in the context 
of a summer research program, we compared the 
ratings of faculty sponsors to naive evaluators. In 
addition to having more knowledge of the student’s 
abilities, faculty sponsors also had a personal 
relationship with the student. A more controlled 
evaluation would compare the ratings of two sets of 
naive evaluators: one that reviewed only the ePortfolio 
and a second that reviewed the student’s work over the 
entire program. We hypothesized and found that for 
most outcomes, faculty sponsors credited students with 
greater learning than naive evaluators. We interpreted 
this to reflect the fact that faculty sponsors had more 
genuine knowledge of students’ abilities. However, it is 
also possible that faculty sponsors credited more 
learning because of the influence of having a personal 
relationship with the student. We believe that this 
interpretation is unlikely for several reasons. First, 
faculty sponsors were not sharing their evaluations with 
the student. This should reduce faculty sponsors’ focus 
on their personal relationship with their students. 
Second, faculty sponsors also served as naive 
evaluators. Serving as a naive evaluator should help the 
faculty sponsor take a neutral perspective when 
considering the work of his/her own students. Third, 
faculty sponsors knew that their students were also 
being assessed by naive evaluators. This knowledge 
should encourage impartiality. Finally, the few 
instances in which faculty sponsors did not credit 
learning that was credited by naive evaluators involved 
outcomes that were the most conceptually difficult. 
This suggests that in these instances faculty sponsors 
used their more extensive knowledge of the student to 
determine that the evidence in the ePortfolio was not 
sufficient to document mastery. It seems unlikely that if 
faculty were biased to credit students because of a 
personal relationship they would exhibit this bias only 
for the less difficult learning outcomes. 

To give students a sense of autonomy in the 
creation of their ePortfolios, we allowed students to 
create their ePortfolios without using a template. We 
believe that this may have increased the extent to which 
students documented and developed academic identity 
by encouraging them to fully engage with the ePortfolio 
as a creative project. However, students’ different 
organizational choices likely made it difficult for some 
evaluators to locate evidence of individual learning 
outcomes. We hypothesize that inter-rater reliability 
was impeded by the unstructured nature of the students’ 
ePortfolios. Were students creating ePortfolios for the 
purpose of receiving credit for prior learning, it would 


be advisable to develop a template that was organized 
by learning outcome and specified the products to be 
included to demonstrate mastery. In contrast, for those 
using ePortfolios as a means of developing and 
documenting academic identity, it may be important to 
allow students more independence in the creation of 
their ePortfolios. 

Miller and Morgaine (2009) found that the 
reflective practices embedded in ePortfolio creation 
helped students to develop academic identity as they 
engaged in complex projects. The use of writing 
prompts (see Appendix) appeared to encourage 
expressions of academic identity in student ePortfolios. 
Our current evidence of academic identity in student 
ePortfolios replicates Singer-Freeman et al. (2014). The 
rubrics tested in this work appear to be a reliable means 
of assessing academic identity. The construct of 
academic identity is similar to those of academic self- 
efficacy and academic goals, which have been found to 
be moderately related to academic persistence (Robbins 
et al., 2004). We believe that academic identity should 
be a central element of student ePortfolios and that the 
evaluation of academic identity in student ePortfolios is 
an important area for future research. 

We found evidence to support our hypothesis that 
ePortfolios would be useful in the evaluation of 
academic identity and applied and collaborative 
learning at the bachelor’s level. As expected, direct 
faculty knowledge of students provided more robust 
evidence of learning and identity than ePortfolios alone. 
Nonetheless, ePortfolio assessment did provide 
evidence of prior learning and identity in the current 
study. Proficiencies similar to those evidenced by our 
students are likely to be present in other high-impact 
activities (e.g., internships, global learning, learning 
communities). The use of rubric-based assessment of 
ePortfolios by trained evaluators in these contexts could 
have similar value. 
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Appendix 

Reflective Writing Prompts 


1. List your favorite 5 “I am” statements (they do not have to match your original ones). Write your goals for 
the summer, the next several years, and the next 10 years. Write about the ways in which the last week has 
been similar to and/or different from other weeks of your life. 

2. List the personal values that you hold. Describe an experience in your life that made you proud of yourself 
and your values. Reflect on the reason that the experience made you proud of your values. Discuss how 
your values fit with the values of the scientific community. 

3. You are now half way done with your summer experience. Please take a look at your goals for the summer 
that you created during the first week. Describe the progress you have made in meeting your goals. Propose 
ways you could increase your progress towards meeting your goals. 

4. Write a reflection on your thoughts about your ideal career. Be sure to address the following questions in 
your response: What skills do you currently have that you think make you well suited for this career? What 
skills might you need to develop further to succeed in this career? Have your thoughts about your ideal 
career changed as the summer has progressed? 

5. Write a reflection on ways that you have and have not changed over the past 5 weeks. If you could do the 
summer program again, what would you do differently and what would you keep the same? 


